Towards Detailed Recognition of Visual Categories

As humans, we have a remarkable ability to perceive the world around us in minute detail purely from the light that is reflected off it – we can estimate material and metric properties of objects, localize people in images, describe what they are doing, and even identify them. Automatic methods for such detailed recognition of images are essential for most human-centric applications and large scale analysis of the content of media collections for market research, advertisement, and social studies. For example, in order to shop for shoes in an on-line catalogue, a system should be able to understand the style of a shoe, the length of its heels, or the shininess of its material. In order to support visual demographics analysis for advertisement, a system should be able to not only identify the people in a scene, but also to understand what kind (style and brand) of clothes they are wearing, whether they are wearing any accessories, and so on. Despite several successes, such detailed recognition is beyond the current computer vision systems. This is a challenging task, and to make progress we have to make advances on several fronts. We need better representations of visual categories that can enable fine-grained reasoning about their properties, as well as machine learning methods that can leverage ‘big-data’ to learn such representations. In order to enable benchmarks for evaluating recognition tasks and to guide learning and inference in models that solve challenging problems, we need to develop better ways of human-computer interaction. My research touches upon several such themes in the intersection of computer vision, machine learning, and human-computer interaction including: